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~ Tho MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

!)□ Responsive to communication(s) filed on . 

2a)K This action is FINAL. 2b)n This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [3 Claim(s) 1-24 is/are pending in the application. 

4a) Of the above claim(s) 1-8 and 15-20 is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

Q)M Claim(s) 9-14 and 21-24 is/are rejected. 
Claim(s) is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10)13 The drawing(s) filed on 18 July 2003 is/are: a)S accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawjng(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (0- 
B)M All b)n Some * c)n None of: 

1 Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) 13 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) 

2) □ Notice of Draflsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 
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DETAILED ACTION 
Election/Restrictions 

1 . Claims 1-8 and 15-20 are withdrawn from further consideration pursuant to 37 CFR 

1 .142(b), as being drawn to a nonelected groups I and III, there being no allowable generic or 
linking claim. Applicant timely traversed the restriction (election) requirement in Paper No. 6. 

Response to Arguments 

2. Applicants arguments with respect to claim 9 on page 11, line 16 to page 12, hne 12 of 
the Remarks (now Paper No. 13) have been considered but are moot in view of the new 
ground(s) of rejection in view of Bergman et al. 

The Applicant alleges that the Examiner used Hindsight to combine Palmer, Itokawa, and 
Hennessey et al on page 12, lines 13-20 of the Remarks (now Paper No. 13). The Examiner 
respectfully disagrees. In response to apphcant's argument that the examiner's conclusion of 
obviousness is based upon improper hindsight reasoning, it must be recognized that any 
judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight 
reasoning. But so long as it takes into account only knowledge which was within the level of 
ordinary skill at the time the claimed invention was made, and does not include knowledge 
gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re 
McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971). Specifically, Itokawa explicitly 
explains that saving both the foreground and background feature data improves the tracking 
(motion) of the object through video in column 7, lines 35-43 and the color histogram of 
Bergman et al fully characterizes the color distribution of a multimedia object further improving 
the tracking. 



Application/Control Number: 09/493 ,0 1 3 Page 3 

Art Unit: 2621 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 9-14, 21, and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Palmer (U.S. Patent No. 5,684,715) in view of Itokawa (U.S. Patent No. 6,404,901), and further 
in view of Bergman et al (U.S. Patent No. 6,564,263). 

Referring to claim 9, 

i. Extracting feature data of a predetermined object and feature data of a background 
area from a frame of an input video is not explicitly explained by Palmer. Palmer does 
explain extracting feature data of a predetermined object in column 6, lines 8-29. The 
attribute generator is responsible for determining characteristics of an object such as 
position, shape, size, and layering information for each video object. However, Palmer 
does not explicitly explain extracting feature data of the background area from a frame. 
Itokawa illustrates extracting both the foreground area and the background area from an 
image input in figure 8, reference numbers 102 and 104. The extraction of feature data 
from the background is explained by Itokawa in column 9, lines 42-50. Itokawa explains 
that saving both the foreground and background feature data improves the tracking 
(motion) of the object through video in column 7, lines 35-43. Furthermore, in order to 
save spatial and temporal information about an entire frame, it would be necessary to 
extract both the object and the background information from a frame. Therefore, it would 
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have been obvious to one of ordinary skill in the art at the time the invention was made to 
extract feature data of a predetermined object and feature data of the background area 
from a frame, as suggested by Itokawa, in the object descriptor of Palmer because the 
object would be completely characterized and tracked more accurately, 
ii. Describing the feature data of the predetermined object and the feature data of the 
background area as a descriptor of the frame, the feature data of the predetermined object 
including a color histogram of an area of the predetermined object, the color histogram 
representing each color element is not explicitly explained by Palmer or Itokawa. Palmer 
does illustrate describing the feature data of a predetermined object as a video descriptor 
in figure 3. Furthermore, Itokawa explains determining the feature data of a background 
area of a frame corresponding to part i of this claim. However, the system of Palmer and 
Itokawa does not include a color histogram as feature data of a predetermined object. 
Bergman et al explain feature descriptors including color histograms in column 8, lines 
42-46 to describe multimedia objects (explained in the abstract). Bergman et al explain 
the possible histogram types and transformations in column 12, line 67 to column 14, line 
2. By fully characterizing the color distribution of a multimedia object and storing the 
characterization (i.e. histogram) in the object descriptor of Palmer and Itokawa, an object 
will be tracked more accurately throughout a series of frames. Therefore, it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to 
describe the feature data of the predetermined object and the background, including a 
color histogram representing each color element of a predetermined object of the frame, 
as suggested by Bergman et al, in the descriptor of Palmer and Itokawa because the 
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complete feature data about an entire frame of video is recorded and object tracking is 
performed more accurately. 

iii. Attaching the descriptor to the frame is illustrated by Palmer in figure 3, as the 

video object descriptor 60 corresponds to object 64 of frame 51 . 

Referring to claim 10, describing a difference between the feature data of the 
predetermined object and the feature data of the background area as the descriptor is not 
explicitly explained by Palmer. However, Itokawa explains in column 9, lines 6-30 that the 
background data and the foreground data are separated, thereby describing the difference 
between the feature data and the background data. Itokawa further illustrates in figure 26, 
illustrates the difference between the background motion vectors (601) and the object motion 
vectors. In order to separate a predetermined object from the background area of a frame, it 
would be necessary to determine a difference between a feature of the object and the 
background. Therefore, it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to describe a difference between the feature data of a predetermined 
object and the feature data of the background area. 

Referring to claim 1 1 , 

i. The feature data of the predetermined object including at least position, outward 
form or size, and moving information of the object is explained by Palmer in column 4, 
lines 51-59 and column 5, lines 7-11. The descriptor is explained to include spatial 
information including the position and size of each object and the temporal information 
includes action information that specifies movement of the object. 
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ii. The feature data of the background area including at least the moving information 
of the background area is not explicitly explained by Palmer. However, Itokawa explain 
that a background area may be moving and that the background movement maybe be 
calculated as the motion value vector, Vb in column 9, lines 21-30 and illustrated in 
figure 1 1 . In order to save information of an entire frame when the background is 
moving, as suggested by Itokawa, it would be necessary to extract both the moving 
information of the object and the moving information of the background. Therefore, it 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to include the moving information of the background in the feature data of the 
background. 

Referring to claim 12, the descriptor including: 

i. A frame number is illustrated by Palmer in figure 4, as each frame has its own 
corresponding frame ID. 

ii. A pointer to a next descriptor is explained by Palmer in column 5, lines 7-11. The 
action information in the descriptors will allow the computer to jump from one frame 
sequence to another frame sequence, which also would contain a descriptor. 

iii. The feature data of the background area corresponds to claim 1 1 . 

iv. The feature data of each object in the frame is explained by Palmer in column 5, 
lines 12-26. Palmer explains that there will be a separate video descriptor for each of the 
objects in the frames. 
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Referring to claim 13, the descriptor being created from a corresponding frame at an 
interval of a predetermined number of frames in the input video is explained by Palmer in 
column 6, lines 23-29 and illustrated in figure 4. The interval of frames in the input video is one. 

Referring to claim 14, this claim includes all the Hmitations of claim 14, but is performed 
on a computer readable medium. Palmer illustrates a computer readable medium capable of 
performing the method of claim 9 in figure 2 as Disk (15). 

Referring to claims 21 and 23, the feature data of the predetermined object including 
average and direction of a moving vector is not expUcitly explained by Palmer. However, 
Itokawa illustrates a moving vector of an object and a background (shaded pixels), including 
average and direction, in figure 18. Therefore, it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to include average and direction of a moving 
vector, as suggested by Itokawa, in the feature data of the predetermined object in the system of 
Palmer because the motion of the object between frames will be defined, thereby improving the 
tracking of the object in video. 

Referring to claims 22 and 24, the feature data of the background including at least one of 
color histogram and camera-work information describing an operation of a camera used for the 
input video is not explicitly explained by Palmer or Itokawa. However, Bergman et al explain 
feature descriptors including color histograms in column 8, lines 42-46 to describe multimedia 
objects (explained in the abstract). By fully characterizing the color distribution of a multimedia 
object and its background and storing the characterization (i.e. histogram) in the object descriptor 
of Palmer and Itokawa, an object will be tracked more accurately throughout a series of frames. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time the invention 
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was made to describe the feature data of the background including a color histogram, as 
suggested by Bergman et al, in the descriptor of Palmer and Itokawa because the complete 
feature data about an entire frame of video is recorded and object tracking is performed more 
accurately. 

Conclusion 

5. The prior art made of record and not reUed upon is considered pertinent to applicant's 
disclosure. 

Vaithilingam et al (U.S. Patent No. 6,41 1,724) - To exhibit using descriptors to 
represent multimedia objects including color histograms as explained in column 10, lines 
19-26. 

6. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, TfflS ACTION IS MADE FINAL, See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 
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7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Hussein Akhavannik whose telephone number is (703)306-4049. 
The examiner can normally be reached on M-F 8:30-5:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H. Boudreau can be reached on (703)305-4706. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Pubhc PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Hussein Akhavannik 
June 18, 2004 





LEO BOUDREAU 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2600 



